Spin-orbit coupling, edge states and quantum spin Hall criticality 
due to Dirac fermion confinement: The case study of graphene 
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We propose a generalized Dirac fermion description for the electronic state of graphene terminated 
by a zigzag edge. This description admits a spin-orbit coupling needed to preserve time-reversal 
invariance of the zigzag confinement, otherwise, for spinless particles, showing the parity anomaly 
typical of quantum electrodynamics in (2+1) dimensions. At a certain critical strength the spin- 
orbit coupling induces a phase transition of the quantum-spin-Hall type. It is manifested by a novel 
type of the edge states consisting of a Kramers' pair of counter-propagating modes with opposite 
spin orientations. Such edge states are capable of accumulating an integer spin in response to a 
transverse electric field in the absence of a magnetic one. They exist without any excitation gap 
in the bulk, due to which our system stands out among other quantum spin Hall systems studied 
earlier. We show that at the transition the local density of states is discontinuous and its energy 
dependence reflects the phase diagram of the system. 

PACS numbers: 73.20.At,73.22.Gk,73.63.Bd 
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Introduction.- Massless Dirac fermions in graphene, 
an isolated two-dimensional (2D) graphite layer, are 
responsible for unconventional electronic properties of 
this materials*, offering new functionalities for nanoelec- 
tronic devices such as recently realized single-electron 
transistors in graphene quantum dots^. This is a typ- 
ical example of a situation where Dirac fermions occur in 
a confined geometry, which brings up the rather general 
issue of the boundary effects in graphene. The need for 
their characterization is one of the outstanding current 
challenges in the field, closely related to the problem of 
Klein tunneling, and, for this reason, having no analogues 
in conventional semiconductors and metals. 

In the present Letter we propose a novel spin-orbit cou- 
pling mechanism originating entirely from Dirac fermion 
confinement rather than being an intrinsic material prop- 
erty. As a model, we consider graphene bounded by a 
zigzag edge and without spin-orbit coupling in the bulk. 
The zigzag boundary is one of the most common types 
of the honeycomb lattice termination (shown in Fig. [1]) 
that stands out due to its unique ability to support edge 
states, decaying in the bulk and delocalized along the 
boundary^ ^^» li ^» . 13 i 14 ' 15 i 16 . The existence of the 
edge states is crucial for the spin-orbit coupling in our 
model. To demonstrate this, we first argue that for spin- 
less electrons the zigzag confinement exhibits an insta- 
bility toward the formation of chiral edge states. This 
would break time-reversal symmetry and drive the sys- 
tem into a state analogous to the integer quantum Hall 
phase proposed by Haldaneii as a realization of the par- 
ity anomaly of (2+l)-dimensional quantum electrody- 
namics (e.g. Ref.|I|). 

We further show that for spin-half particles the re- 
quirement for the cancellation of the parity anomaly im- 
plies the coupling between the spin and orbital degrees 
of freedom: Instead of the chiral edge we obtain a pair of 
counter-propagating gapless edge modes with the oppo- 
site spin orientations. They are related to each other by 
time-reversal symmetry and have a Kramers' degenerate 



spectrum, ensuring their robustness against time-reversal 
invariant perturbations. 

Such a Kramers' doublet of gapless edge states is a 
characteristic signature of the quantum spin Hall (QSH) 
systems (e.g. Refs. 19.20,21.22). The recent interest in 
these systems is motivated by the principal possibility 
to realize a time-reversal invariant integer quantum Hall 
state in which the spin Hall conductance is quantized. 
The quantization is due to the spin accumulation ability 
of the edge states that does not require a strong magnetic 
field. The transition into the QSH state discussed in the 
literatur o 19 i 20 ' 21 i 22 is accompanied by opening a finite ex- 
citation gap in the 2D bulk generated, e.g. by spin-orbit 
interactions. In this respect, our case is special since 
zigzag-terminated graphene (as well as the unbounded 
system) is a zero-gap semiconductor, and the spin-orbit 
coupling at the edge does not influence the bulk elec- 
tronic states. Nonetheless, we show that the system does 
exhibit a transition from the ordinary zero-gap semicon- 
ductor phase with spin-degenerate edge states*^ to the 
novel phase possessing a Kramers' pair of gapless spin- 
filtered edge modes, which can be identified as a QSH 
state. 

The absence of the bulk energy gap makes it difficult to 
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FIG. 1: Geometry: system occupies the region \x\ < L/2, y > 
with a zigzag edge at y = described by the boundary 
condition, Eq. {J}, and is periodic in x direction, ng is the 
boundary normal. 
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characterize the QSH state by electric transport means. 
We suggest an alternative method based on the tunneling 
spectroscopy of the local density of states (LDOS). In the 
QSH and ordinary states, the LDOS shows drastically 
different energy dependences that cannot be continuously 
transformed into each other, implying that the two states 
are topologically distinct. 

Discrete symmetries of the problem.- We start 
with the spinless model containing a pair of Weyl 
fcrmions originating from the two inequivalent valleys of 
graphene's Brillouin zonei 8 -. They can be represented by 
a four-component function ip satisfying the Dirac equa- 
tion etp — —ihv(r z ® S) V"0 with energy e measured from 
the Fermi level and momentum —ifiV confined to the 
graphene plane, see Fig. [T] (v is the Fermi velocity). We 
consider a single edge (along the x-axis) that does not 
cause inter-valley scattering, which can be described by 
an effective boundary condition of Ref. 

V> = Mip\ y=0 , 
M = cos(Ar z + Ct ) ® E^ + sin(Ar z + (t ) ® (1) 

The Pauli matrices Yt x ,z act in sublattice (pseudospin) 
space, while t z and the unit matrix To operate in valley 
space. Equation (fT]) contains two parameters, £ and A, 
that in the limit C -> 0, A -> § yield M = t z <g> £ z which 
is the continuum model for a zigzag graphene edg o 9 ' 16 . 
We intend to study the stability of the zigzag edge states 
with respect to small deviations from the zigzag bound- 
ary condition. To introduce such deviations we use the 
general formula for M ([1]), assuming small but finite pa- 
rameters 



£<1, A = A-tt/2<1. 



(2) 



They explicitly violate 2D parity and inversion symme- 
tries of the system, respectively, allowing us to study 
their interplay, which has not been done previously 8 - ' 16 ' 24 . 

The 2D parity of the Dirac equation is defined as co- 
ordinate reflection along the edge, x — > — x accompanied 
by a unitary spinor transformation^ 8 , 



ip(x,y) -> Vi/j(-x,y), V = 



£z- 



(3) 



The 2D inversion is realized by an in-plane rotation by 
7r, yielding x, y — > —x, —y, along with the spinor trans- 
formation 



ip(x,y) — >Iip(—x, -y), Z = t ®£ z . 



(4) 



The boundary condition does not share these symmetries 
since both M v = VMV^ and M 1 = TMtf differ from 
M in Eq. (fT|): 

M v = cos(At z - Ct ) ® E x + sin(Ar z - Ct ) ® £ 2 ,(5) 
M 1 = - cos(Ar 2 + Ct ) ® S K + sin(Ar 2 + Ct ) ® S z (6) 

Setting ( — > restores the parity (M v — > M), and if, 
additionally, A — > the inversion symmetry is recovered 
as well (M 1 M). 



We note that for £ ^ the 2D parity is broken simul- 
taneously with the time-reversal (T) symmetry as the 
latter is represented by the operator T = VC [16| where 
C is complex conjugation. Despite the violation of the 
T symmetry, the use of Eq. ([I]) is justified because T is 
only an effective symmetry of spinless particles (no spin 
degrees of freedom involved so far). Apart from that, the 
T symmetry breaking is considered as a weak perturba- 
tion with C ^ 1- The small parameters Q and A compete, 
and, therefore, it is natural to carry out the stability 
analysis as a two-parametric problem. This analysis will 
eventually lead us to the time-reversal invariant bound- 
ary condition (|23p . which is behind the main results of 
this study, e.g. the prediction of the quantum spin Hall 
transition. 

Spinless edge states and parity anomaly.- Let us 

first understand the implications of the broken symme- 
tries for the equilibrium properties of spinless electrons 
which can be described by the LDOS, 



V(e,r) = -(l/7r)ImTrG fl (r,r), 



(7) 



where the trace of the retarded matrix Green's function 
G R is taken in r ® E space. G R satisfies the Dirac equa- 
tion, 

£ G R (r, r') + ihv{r z ® S)VG R (r, r') = 6(r - r'), (8) 

the boundary condition ([T]), and G R \ y ^oo < oo. We also 
assume a periodic boundary condition in the x direction 
(with period L), modulated by a magnetic phase <f>: 



G r \ x= l/2 — G R \ x= _ L / 2 c 



(9) 



G R can be expanded in plane waves e nX with 



k n = ^-{n + c^), ne Z(0,±1,...), (10) 

following from Eq. ([9]). Then, the matrix Dirac equation 
can be reduced to ordinary differential equations for the 
diagonal elements of G R which can be readily solved. The 
final result is 



G R (r,r')= Yl 



Tn + TT Z \ / rflV „ 

- » ® E + — EV 



T=±l,n£Z 

x (G+ kn (y,y')Eo + G- k Jy,y')2l z ) ^ * * , (11) 



^(»')^(^ W) - e " bl " 1 ) 



q n + k n s T e -q n { y +y') 
2(e — TC T Hvk n ) 



(12) 



G~ (v v') ~ k n+1nS T -rc T e/tlv (yW) 

G rkAy^y)- 2 (e-Tc T hvk n ) 6 
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Here r = ±1 labels the valleys and q n — \/k 2 — e 2 /h 2 v 2 . 
The edge-state spectrum follows from Eq. (fT2|) taken at 
e — > TC T hvk n : 



Gt kn (y,y') 



k n S T Q(k n S T ) e -k n s T (y+y') 

e — Tc T hvk n 



(14) 



It has a pole only if the unit step function Q(k n s T ) is 
nonzero, which determines the spectrum as 



e T \k n 



TC T hvk n for k n s T > 0, 



(15) 



with c T = cos(Ar + C) and s T — sin(Ar+£). It is particle- 
hole asymmetric due to the broken discrete symmetries. 

Using Eq. pip and the Poisson summation formula we 
can express the LDOS as 



T.neZ 



(16) 



tt2 E 



-27Tincp 



r.neZ 



dke mkL ImGt k (y,y), 



where the integration over the edge (|fc| > \e\/hv) and 
bulk (|fc| < \e\/hv) states should be done separately. Ac- 
cordingly, the LDOS V(e, y) = V e (e, y)+Vb(e, y) contains 
the edge contribution, 



V e (e,y) = — \2whvc T \ 1 0(ets t c t ) x 

T,n£Z 
( eL 



(17) 



x exp in 

\nvc T 

and the bulk one, 
Vb(e,y) 



— 2tt<J) I d y exp 



hv 



ir 2 h 2 v 2 



E 



tt/2 



C?7 X 



(18) 



T,nEZ 

cos(27rn0) cos(nk e L cos 7) 



cos 7 x 



.4 cos (fc7 sin 7) 



rs T c T sin 7 sin ( sin 7) 



c 2 sin 2 7 



+ sin(27m</>) sin(nfc e L cos 7) sin 7 cos 7 x 
rc T sin 7 cos ( -jp sin 7) — s T sin ( ^p sin 7) 



s 2 . + c 2 . sin 7 

The edge LDOS (jTTJ) vanishes for s T = 0, i.e. for in- 
plane pseudospin orientation at the edge [see, Eq. (JTJ)] . 
This points to the topological origin of the edge states, 
since their existence requires a nontrivial 3D pseudospin 
structure with the out-of-plane component E z . 

If we now restore the symmetries, setting £, A — > 
(i.e. c T — > and s r — ► r), the bulk LDOS (|18jl recovers 
the particle- hole symmetry, while the edge one, Eq. (jTTJ) , 
fails to do so because of the singularity at c T = 0. This 
anomalous asymmetry leads to a finite charge density, 



p(y) 



de[V(e,y)-V(-e,y)]/2, (19) 



entirely localized at the edge, since the symmetric bulk 
LDOS cancels out. It can be calculated from Eq. (fl7|l as 



p(y) 



cosh 



^F(V^)]l>gn(A + TC) 



sinh 



~TT d y~ 



sinh(27r?//L) 
^(^-^)]E^gn(Ar 







sinh(27ry/L) 



(20) 



where the small parameters £ and A enter through the 
sign function, and the dimensionless magnetic flux is con- 
fined to the interval < \<f>\ < 1 due to periodicity of 
Eq. The net edge charge, Q = L f Q dyp(y) contains 
a flux-dependent contribution given by 



Q(4>) = Ne U 



sgn< 



N 



\ J2 Bgn(Ar- C).(21) 



T=±l 



Remarkably, adiabatic sweeping of the flux from to 1 
leads to the accumulation of the integer charge AQ = 
Q(l) - Q(0) = with N = 0,±1. The nontrivial in- 
tegers N = ±1 require |C| > |A|. This is the condition 
for the formation of a chiral Dirac fermion edge channel, 
which follows from the edge-state spectrum, Eq. fTJ 
linearized with respect to £ and A, 



e T (k n ) = —(At + Qhvk n for k n r > 0. 



(22) 



We see that for \(\ > | A| , the two graphene valleys prodive 
a pair of Weyl fermion edge states propagating in the 
same direction, i.e. a single chiral Dirac fermion edge 
channel. 

The interpretation of Eq. (|2"Tj) is quite straightfor- 
ward if we notice that the periodicity of our system [due 
to Eq. ([9|)] is that of a cylinder enclosing a magnetic 
flux <j) (in units of ch/e). Repeating Laughlin's argu- 
ment 2 ^ one can identify AQ as the Hall charge accu- 
mulated in the chiral edge channel in response to the 
electric field E x = —(h/eL)<j> generated by the adiabat- 
ically varying flux. On the other hand, since there is 
no quantizing magnetic field, our result is close in spirit 
to Haldane's integer quantum Hall effect^ realizing the 
so-called parity anomaly 18 . The anomaly consists in 
the odd (^-dependence of Q, which is in sharp contrast 
to the common expectation that the electric charge (a 
scalar quantity) should be an even function of the mag- 
netic flux. Equation (|21[) has nevertheless normal par- 
ity. Indeed, the parity operation, Eq. ([3]) is equivalent 
to Cj —> — C ~4> which does not affect Q. Thus, the 
N = ±1 quantum Hall phases are characterized by the 
number of the chiral modes (|iV| = 1) and two possible 
propagation directions (±). 

The parity anomaly indicates a magnetic instability 
of the continuum zigzag-edge model with respect to the 
small T symmetry breaking perturbation. Importantly, 
for A = it takes only an infinitesimally small perturba- 
tion £ to drive the system into the quantum Hall state 
with TV = sgn£. The conventional zero-gap semicon- 
ductor state (i.e. N = 0) is recovered when £ = and 
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A is finite. The latter could originate from a staggered 
boundary potentiate. 

While the parity anomaly and its connection to the 
quantum Hall physics are of interest on their own 
grounds, in what follows we return to the T symmetric 
situation. The T invariance is recovered in the presence 
of two spin- 1/2 fermion species experiencing opposite- 
sign quantum Hall effects. As the spatial 2D parity 
remains broken, we expect a confinement-induced spin- 
orbit coupling with the strength controlled by (. It could 
for instance be induced by boundary strain. Unfortu- 
nately, we do not have the means to calculate the cou- 
pling strength £ on the microscopic basis. Such a situa- 
tion is quite common for the theory of spin-orbit interact- 
ing systems 2 ^, and has hardly been a serious obstacle for 
predicting new electronic properties and functionalities. 
This is also the main objective here. 

Edge spin-orbit coupling and QSH transition. - 
Let us combine the pseudospinors for opposite spin pro- 
jections | and I (e.g. on z axis) into a single function 
\& = (V't, ipi) an d consider the following generalization of 
the boundary condition, Eq. |T]): * = , 

M = cos(Ao- ® t z + (<r z <g> r ) ® T, x + 

+ s,m(A(T (Z>T z +(<7 z (g)T )(g>T, z , (23) 

where ao,a z are unit and Pauli matrices in spin space. 
The presence of a z makes the new boundary condi- 
tion invariant under the time reversal operation T = 
i<jy (g) t x (g) Y> x C where C is the complex conjugation. This 
also follows from the new edge spectrum obtained from 
Eq. (j22|) by £ — > £er where a = ±1 are the eigenvalues of 



-(At + (,a)hvk n 



for k n r > 0. 



(24) 



In the presence of the spin-orbit term (cx £) Eq. ex- 
hibits Kramers' degeneracy e_ 7 - ! _ cr (— k n ) = e Tt<T (k n ) as 
a manifestation of the time-reversal symmetry. Equa- 
tion (|23|) can be recast in the vector form suitable for 
more complicated edge geometries: 

M = cos A t q <g> N (g) £ + sin A t z ® (n B x N) <g> £, 

N = |(e - zz)e< £er + |(e + i^e"^ ", (25) 

where e is a unit vector pointing along the edge, z is a 
unit vector normal to the plane, and the components of 
the vector N are operators acting in spin space. 

The two spin subsystems separately form the quan- 
tum Hall states with the integer factors and ob- 
tained from Eq. (f2Tj) by substitution £ — > (cr. How- 



ever, there is no net Hall charge since N-t 



Nt 



0. 



Instead, from Eq. (|21j) we find the nonzero Hall spin, 
5(0) = (ft/2e)(Q T (0) - Q|(0)), or, explicitly, 



5(0) = /V7i 



sgn< 



A=^-^. (26) 



Here N((,X) is given by Eq. (f2"Tj) . It is a singular func- 
tion of the symmetry breaking parameters which deter- 
mines the phase diagram of the system shown in Fig. [5] 
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FIG. 2: Phase diagram in space of parameters £ and A break- 
ing 2D parity and inversion, respectively. TV = and TV = ±1 
label ordinary zero-gap semiconductor and quantum spin Hall 
states, respectively. The critical point £ = A = corresponds 
to the parity- and inversion-symmetric zigzag edge. Panels 
(a), (b) and (c) show the edge-state spectrum, Eq. (124[1 . for 
r = ±l. 



The phases with N = correspond to ordinary zero- 
gap semiconductor states with zero spin accumulation 
AS = 5(1) -5(0) = 0, while the phases with N = ±1 can 
be identified as quantum spin Hall (QSH) st&tes^i2!LZL£i 
in which AS = ±h. This identification is supported by 
the fact that the two valleys, r = ±1, provide a pair 
of spin-dependent edge states related by time-reversal 
symmetry [panel (c) in Fig. |2J. Within such a pair the 
intervalley backscattering is forbidden unless there are 
time-reversal noninvariant interactions mixing the valleys 
and spin projections. In contrast, in the ordinary zero- 
gap semiconductor (£ = and TV = 0) the edge states 
are spin degenerate [panel (a) in Fig. [2], and, therefore, 
spin-independent intervalley backscattering can lead to 
Anderson localization- 2 -. This is, for instance, the case 
for the edge states originally predicted b y F ujita et al^ 
and their later generalizations (e.g. Ref . . Also, un- 
like QSH topological insulator a 19 ' 20 i 21 ' 22 , in our case the 
edge states exist without any spin-orbit bulk energy gap, 
since they are supported by the nontrivial pseudospin 
structure of the zigzag boundary. The novelty of this 
type of edge states is most prominently manifested by 
the nonvanishing spin accumulation, 



5(0 ->0) = -iV(/y 2)sgn0, 



(27) 



resulting from the zero mode n = in Eq. (|17[) . 

Because of the gapless 2D bulk, transport measure- 
ments are hardly suitable for the characterization of the 
TV = ±1 and TV — phases. We propose a more ro- 
bust method based on the tunneling spectroscopy of the 
LDOS. FigureHshows V(e,y) = £ ff [VJ(e, y) + V£(e, y)], 
where V° b (c,y) are given by Eqs. (fTT)) and (fTS|) with 
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FIG. 3: LDOS at representative points of phase diagram in 
Fig. [2] (a) N = phase: ( = 0, A = 0.4, (b) close to transition 
from N = side: £ = 0.15, A = 0.25, (c) close to transition 
from N = -1 side: £ = 0.3, A = 0.2, and (d) N = -1 phase: 
£ = 0.3, A = 0. We took the limit L — > oo in which only the 
n = terms in Eqs. (JT7| and (JTHJ) contributed; eo = hv/2y 
and Vo = 1/hvy. 



C — > C "- Panels (a) and (d) correspond to the N = and 
AT = — 1 phases, respectively, which differ by the sym- 
metry of the energy dependence of V(e, y). The particle- 
hole asymmetric LDOS of the N = phase transforms 
into the symmetric LDOS of the N = — 1 phase through 
a discontinuity [Figs. [3Jb) and (c)] coming from zero- 
energy states existing on the critical lines of the diagram 
in Fig. [21 The discontinuous phase transition implies 
that the N = ±1 states are topologically distinct from 
an ordinary zero-gap semiconductor with N = 0. 

Conclusions.- Within the continuum model for 
zigzag-terminated graphene, we have demonstrated the 
possibility of confinement-mediated spin-orbit coupling. 
The system exhibits a phase transition into a quantum 
spin Hall state in the sense that it possesses a Kramers' 
doublet of spin-dependent edge states, accumulating an 
integer spin, albeit there is no true gap in the system's 
bulk as in the usual quantum spin Hall systems. We show 
that the local density of states can be used to distinguish 
the quantum spin Hall state from ordinary zero-gap semi- 
conductor state of zigzag-terminated graphene. Our find- 
ings also imply that zigzag graphene edges can be spin- 
active without interaction-induced magnetismi i 27 ' 28 i 29 . 
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son and A. D. Mirlin for discussions. The work was sup- 
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Research Foundation (DFG). 
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